Skip to content

feat(envs): add RoboCerebra long-horizon manipulation benchmark#3314

Open
pkooij wants to merge 2 commits intofeat/benchmark-cifrom
feat/robocerebra-benchmark
Open

feat(envs): add RoboCerebra long-horizon manipulation benchmark#3314
pkooij wants to merge 2 commits intofeat/benchmark-cifrom
feat/robocerebra-benchmark

Conversation

@pkooij
Copy link
Copy Markdown
Member

@pkooij pkooij commented Apr 8, 2026

Summary

  • Adds new robocerebra env type wrapping LIBERO's libero_10 suite with RoboCerebra-specific defaults (256×256, 20 FPS, camera keys image/wrist_image matching the HF dataset)
  • Fixes a gap in the LIBERO factory: camera_name_mapping was accepted by LiberoEnv.__init__ but never forwarded through create_libero_envs → _make_env_fns → _make_env; now correctly propagated end-to-end (also benefits existing LiberoEnv users who pass --env.camera_name_mapping via CLI)
  • Adds robocerebra optional dependency group in pyproject.toml (aliases lerobot[libero])

Dataset

The dataset CollisionCode/RoboCerebra_lerobot_v3.0 is already in LeRobot v3.0 format — no conversion needed.

Camera keys in the dataset match our defaults:

  • observation.images.image (agent-view)
  • observation.images.wrist_image (wrist)

Files changed

File Change
src/lerobot/envs/configs.py Add RoboCerebraEnv; fix LiberoEnv.create_envs to forward camera_name_mapping
src/lerobot/envs/libero.py Thread camera_name_mapping through create_libero_envs → _make_env_fns → _make_env
pyproject.toml Add robocerebra extra (= lerobot[libero], Linux only)
tests/envs/test_robocerebra_env.py 10 unit tests (no LIBERO install needed — factory fully mocked)
docs/source/robocerebra.md Install, dataset, task table, eval commands, config reference, citation

Test plan

  • 10 unit tests pass (pytest tests/envs/test_robocerebra_env.py) — no LIBERO needed, all env creation mocked
  • pre-commit run -a passes on all changed files
  • Eval smoke test (requires Linux + LIBERO + GPU):
    # Test existing libero path (user-provided command)
    lerobot-eval \
        --policy.path=pepijn223/smolvla_libero \
        --env.type=libero \
        --env.task=libero_spatial \
        --eval.batch_size=1 \
        --eval.n_episodes=1 \
        --eval.use_async_envs=false \
        --policy.device=cuda \
        '--env.camera_name_mapping={"agentview_image": "camera1", "robot0_eye_in_hand_image": "camera2"}' \
        --policy.empty_cameras=1
    
    # Test new robocerebra env type
    lerobot-eval \
        --policy.path=pepijn223/smolvla_libero \
        --env.type=robocerebra \
        --env.task=libero_10 \
        --eval.batch_size=1 \
        --eval.n_episodes=1 \
        --eval.use_async_envs=false \
        --policy.device=cuda

Publishing proposal

The dataset is already on HuggingFace at CollisionCode/RoboCerebra_lerobot_v3.0. To mirror it under the lerobot/ org:

huggingface-cli repo create robocerebra --type dataset --organization lerobot
huggingface-cli upload lerobot/robocerebra CollisionCode/RoboCerebra_lerobot_v3.0 --repo-type dataset

Requires lerobot/ org write access — can be done separately after merge.

🤖 Generated with Claude Code

@pkooij pkooij force-pushed the feat/robocerebra-benchmark branch from 865c2a1 to cc8c571 Compare April 8, 2026 14:29
@pkooij pkooij changed the base branch from feat/async-vector-env to feat/benchmark-ci April 8, 2026 14:29
@pkooij pkooij force-pushed the feat/benchmark-ci branch from e89e6d9 to 927118e Compare April 8, 2026 17:22
pkooij and others added 2 commits April 9, 2026 10:05
Introduces a new `robocerebra` environment type that wraps the LIBERO
libero_10 suite with RoboCerebra-specific defaults (256×256 resolution,
20 FPS, camera keys `image`/`wrist_image` matching the HF dataset).

Also fixes a gap in the LIBERO factory chain: `camera_name_mapping` was
accepted by `LiberoEnv.__init__` but never forwarded through
`create_libero_envs` → `_make_env_fns` → `_make_env`. Both `LiberoEnv`
and `RoboCerebraEnv` now propagate it correctly end-to-end.

- `envs/configs.py`: `RoboCerebraEnv` config + `LiberoEnv.create_envs` fix
- `envs/libero.py`: thread `camera_name_mapping` through factory chain
- `pyproject.toml`: `robocerebra` optional dep group (= `lerobot[libero]`)
- `tests/envs/test_robocerebra_env.py`: 10 unit tests (no LIBERO needed)
- `docs/source/robocerebra.md`: install, dataset, eval commands, citation

Dataset: CollisionCode/RoboCerebra_lerobot_v3.0 (already LeRobot v3 — no conversion needed)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds the RoboCerebra benchmark to the benchmark integration test suite
introduced in #3309. Follows the same pattern: one isolated Docker image
per benchmark so dependency trees cannot collide.

- docker/Dockerfile.benchmark.robocerebra: installs lerobot[robocerebra]
  only (= lerobot[libero] alias: hf-libero + dm-control + mujoco)
- .github/workflows/benchmark_tests.yml: full workflow with libero,
  metaworld, and robocerebra parallel jobs; robocerebra job builds its
  own image and runs a 1-episode smoke eval on libero_10

Note: benchmark_tests.yml is also created in #3309. Whichever PR merges
second will need a trivial conflict resolution (add the robocerebra job
block to the existing file).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@pkooij pkooij force-pushed the feat/robocerebra-benchmark branch from cc8c571 to 0a39e51 Compare April 9, 2026 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant